Scalable federated learning for emergency care using low cost microcomputing: Real-world, privacy preserving development and evaluation of a COVID-19 screening test in UK hospitals (preprint)

Background Tackling biases in medical artificial intelligence requires multi-centre collaboration, however, ethical, legal and entrustment considerations may restrict providers' ability to participate. Federated learning (FL) may eliminate the need for data sharing by allowing algorithm development across multiple hospitals without data transfer. Previously, we have shown an AI-driven screening solution for COVID-19 in emergency departments using clinical data routinely available within 1h of arrival to hospital (vital signs & blood tests; CURIAL-Lab). Here, we aimed to extend and federate our COVID-19 screening test, demonstrating development and evaluation of a rapidly scalable and user-friendly FL solution across 4 UK hospital groups. Methods We supplied a Raspberry Pi 4 Model B device, preloaded with our end-to-end FL pipeline, to 4 NHS hospital groups or their locally-linked research university (Oxford University Hospitals/University of Oxford (OUH), University Hospitals Birmingham/University of Birmingham (UHB), Bedfordshire Hospitals (BH) and Portsmouth Hospitals University (PUH) NHS trusts). OUH, PUH and UHB participated in federated training and calibration, training a deep neural network (DNN) and logistic regressor to predict COVID-19 status using clinical data for pre- pandemic (COVID-19-negative) admissions and COVID-19-positive cases from the first wave. We performed federated prospective evaluation at PUH & OUH, and external evaluation at BH, evaluating the resultant global and site-tuned models for admissions to the respective sites during the second pandemic wave. Removable microSD storage was destroyed on study completion. Findings Routinely collected clinical data from a total 130,941 patients (1,772 COVID-19 positive) across three hospital groups were included in federated training. OUH, PUH and BH participated in prospective federated evaluation, with sets comprising 32,986 patient admissions (3,549 positive) during the second pandemic wave. Federated training improved DNN performance by a mean of 27.6% in terms of AUROC when compared to models trained locally, from AUROC of 0.574 & 0.622 at OUH & PUH to 0.872 & 0.876 for the federated global model. Performance improvement was more modest for a logistic regressor with a mean AUROC increase of 13.9%. During federated external evaluation at BH, the global DNN model achieved an AUROC of 0.917 (0.893-0.942), with 89.7% sensitivity (83.6-93.6) and 76.7% specificity (73.9- 79.1). Site-personalisation of the global model did not give a significant improvement in overall performance (AUROC improvement <0.01), suggesting high generalisability. Interpretations We present a rapidly scalable hardware and software FL solution, developing a COVID-19 screening test across four UK hospital groups using inexpensive micro- computing hardware. Federation improved model performance and generalisability, and shows promise as an enabling technology for deep learning in healthcare. Funding University of Oxford Medical & Life Sciences Translational Fund/Wellcome

Serology assays used in SARS-CoV-2 seroprevalence surveys worldwide: a systematic review and meta-analysis of assay features, testing algorithms, and performance (preprint)

Background: Many SARS-CoV-2 serological assays were rapidly developed during the COVID-19 pandemic. However, differences in detection mechanism limit the comparability of assay outputs. Methods: As part of the SeroTracker global living systematic review of SARS-CoV-2 seroprevalence studies, we collated serological assays used in serosurveys between January 1, 2020 and November 19, 2021. We mapped performance metrics to the manufacturer, third-party head-to-head, and independent group evaluations, comparing the assay performance data using a mixed-effect beta regression model. Results: Among 1807 serosurveys, 192 distinctive commercial assays and 380 self-developed assays were identified. According to manufacturers, 28.6% of all commercial assays met WHO criteria for emergency use (sensitivity [Sn.] >= 90.0%, specificity [Sp.] >= 97.0%). Third-party and independent evaluations indicated that manufacturers overstated the Sn. of their assays by 5.4% and 2.8%, and Sp. by 6.3% and 1.2%. We found in simulations that inaccurate Sn. and Sp. can substantially bias seroprevalence estimates corrected for assay performance. Conclusions: The Sn. and Sp. of the serological assay are not fixed properties, but varying features depending on testing population. To achieve precise population estimates and to ensure comparability, serosurveys should select assays with strong, independently validated performance and adjust seroprevalence estimates based on assured performance data.

RapiD_AI: A framework for Rapidly Deployable AI for novel disease & pandemic preparedness (preprint)

COVID-19 is unlikely to be the last pandemic that we face. According to an analysis of a global dataset of historical pandemics from 1600 to the present, the risk of a COVID-like pandemic has been estimated as 2.63% annually or a 38% lifetime probability. This rate may double over the coming decades. While we may be unable to prevent future pandemics, we can reduce their impact by investing in preparedness. In this study, we propose RapiD_AI: a framework to guide the use of pretrained neural network models as a pandemic preparedness tool to enable healthcare system resilience and effective use of ML during future pandemics. The RapiD_AI framework allows us to build high-performing ML models using data collected in the first weeks of the pandemic and provides an approach to adapt the models to the local populations and healthcare needs. The motivation is to enable healthcare systems to overcome data limitations that prevent the development of effective ML in the context of novel diseases. We digitally recreated the first 20 weeks of the COVID-19 pandemic and experimentally demonstrated the RapiD_AI framework using domain adaptation and inductive transfer. We (i) pretrain two neural network models (Deep Neural Network and TabNet) on a large Electronic Health Records dataset representative of a general in-patient population in Oxford, UK, (ii) fine-tune using data from the first weeks of the pandemic, and (iii) simulate local deployment by testing the performance of the models on a held-out test dataset of COVID-19 patients. Our approach has demonstrated an average relative/absolute gain of 4.92/4.21% AUC compared to an XGBoost benchmark model trained on COVID-19 data only. Moreover, we show our ability to identify the most useful historical pretraining samples through clustering and to expand the task of deployed models through inductive transfer to meet the emerging needs of a healthcare system without access to large historical pretraining datasets.

Timeliness of reporting of SARS-CoV-2 seroprevalence results and their utility for infectious disease surveillance (preprint)

Seroprevalence studies have been used throughout the COVID-19 pandemic to monitor infection and immunity. These studies are often reported in peer-reviewed journals, but the academic writing and publishing process can delay reporting and thereby public health action. Seroprevalence estimates have been reported faster in preprints and media, but with concerns about data quality. We aimed to (i) describe the timeliness of SARS-CoV-2 serosurveillance reporting by publication venue and study characteristics and (ii) identify relationships between timeliness, data validity, and representativeness to guide recommendations for serosurveillance efforts. We included seroprevalence studies published between January 1, 2020 and December 31, 2021 from the ongoing SeroTracker living systematic review. For each study, we calculated timeliness as the time elapsed between the end of sampling and the first public report. We evaluated data validity based on serological test performance and correction for sampling error, and representativeness based on use of a representative sample frame and adequate sample coverages. We examined how timeliness varied with study characteristics, representativeness, and data validity using univariate and multivariate Cox regression. We analyzed 1,844 studies. Median time to publication was 154 days (IQR 64-255), varying by publication venue (journal articles: 212 days, preprints: 101 days, institutional reports: 18 days, and media: 12 days). Multivariate analysis confirmed the relationship between timeliness and publication venue and showed that general population studies were published faster than special population or health care worker studies; there was no relationship between timeliness and study geographic scope, geographic region, representativeness, or serological test performance. Seroprevalence studies in peer-reviewed articles and preprints are published slowly, highlighting the limitations of using the academic literature to report seroprevalence during a health crisis. More timely reporting of seroprevalence estimates can improve their usefulness for surveillance, enabling more effective responses during health emergencies.

Machine Learning Generalizability Across Healthcare Settings: Insights from multi-site COVID-19 screening (preprint)

As patient health information is highly regulated due to privacy concerns, the majority of machine learning (ML)-based healthcare studies are unable to test on external patient cohorts, resulting in a gap between locally reported model performance and cross-site generalizability. Different approaches have been introduced for developing models across multiple clinical sites, however no studies have compared methods for translating ready-made models for adoption in new settings. We introduce three methods to do this - (1) applying a ready-made model as-is; (2) readjusting the decision threshold on the output of a ready-made model using site-specific data; and (3) finetuning a ready-made model using site-specific data via transfer learning. Using a case study of COVID-19 diagnosis across four NHS Hospital Trusts, we show that all methods achieve clinically-effective performances (NPV >0.959), with transfer learning achieving the best results (mean AUROCs between 0.870-0.925). Our models demonstrate that site-specific customization improves predictive performance when compared to other ready-made approaches.

Algorithmic Fairness and Bias Mitigation for Clinical Machine Learning: Insights from Rapid COVID-19 Diagnosis by Adversarial Learning (preprint)

Machine learning is becoming increasingly promi- nent in healthcare. Although its benefits are clear, growing attention is being given to how machine learning may exacerbate existing biases and disparities. In this study, we introduce an adversarial training framework that is capable of mitigating biases that may have been acquired through data collection or magnified during model development. For example, if one class is over-presented or errors/inconsistencies in practice are reflected in the training data, then a model can be biased by these. To evaluate our adversarial training framework, we used the statistical definition of equalized odds. We evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments, and aimed to mitigate regional (hospital) and ethnic biases present. We trained our framework on a large, real-world COVID-19 dataset and demonstrated that adversarial training demonstrably improves outcome fairness (with respect to equalized odds), while still achieving clinically-effective screening performances (NPV>0.98). We compared our method to the benchmark set by related previous work, and performed prospective and external validation on four independent hospital cohorts. Our method can be generalized to any outcomes, models, and definitions of fairness.

SeroTracker-ROB: reproducible decision rules for risk of bias assessment of seroprevalence studies (preprint)

BackgroundEvaluating seroprevalence study risk of bias (RoB) is crucial for robust infection surveillance, but can be a time-consuming and subjective process. We aimed to develop decision rules for reproducible RoB assessment and an automated tool to implement these decision rules. MethodsWe developed the SeroTracker-RoB approach to RoB assessment. To do so, we created objective criteria for items on the Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Prevalence Studies and developed decision rules for RoB based on these items. The criteria and decision rules were based on published guidance for assessing RoB for prevalence studies and expert opinion. Decision rules were validated against the SeroTracker database of seroprevalence studies, which included consensus manual RoB judgements from two independent reviewers. We measured efficiency by calculating paired-samples t-test for time to judge RoB using the automated tool versus manually for 25 randomly selected articles from the SeroTracker database, coverage as the proportion of database studies where the decision rules could evaluate RoB, and reliability by calculating intraclass correlations between automated and manual RoB assessments. ResultsWe established objective criteria for seven of nine JBI items. We developed a set of decision rules with 61 branches. The SeroTracker-RoB tool was significantly faster than manual assessment with a mean time of 0.80 vs. 2.93 minutes per article (p<0.001), classified 100% (n = 2,070) of studies, and had good reliability compared to manual review (intraclass correlation 0.77, 95% confidence interval 0.74 to 0.80). The SeroTracker-RoB Excel Tool embeds this approach in a simple data extraction sheet for use by other researchers. ConclusionsThe SeroTracker-RoB approach was faster than manual assessment, with complete coverage and good reliability compared to two independent human reviewers. This approach and tool enable rapid, transparent, and reproducible evidence synthesis of infection prevalence studies, and may support public health efforts during future outbreaks and pandemics. O_TEXTBOXWhat is new? O_LIWhat is already known: Risk of bias assessments are a core element of evidence synthesis but can be time consuming and subjective. As such, there is a need for validated and transparent tools to automate such assessments, particularly during disease outbreaks and pandemics to inform public health decision making. However, there are currently no automated tools for risk of bias assessment of prevalence studies. C_LIO_LIWhat is new: We developed a reproducible approach to risk of bias assessment for SARS-CoV-2 seroprevalence studies. The automated approach was five times faster than manual human assessment, successfully categorized all 2,070 studies that it was tested on, and had good agreement with manual review. We built a simple Excel tool so that other researchers can use this automated approach. C_LIO_LIPotential impact: The SeroTracker-RoB approach and tool enables rapid, transparent, and reproducible risk of bias assessments for SARS-CoV-2 seroprevalence studies, and could be readily adapted for other types of disease prevalence studies. This process may also be applicable to automation of critical appraisal and risk of bias assessment for other types of studies and in other scientific disciplines. C_LI C_TEXTBOX

Real-world evaluation of AI driven COVID-19 triage for emergency admissions: External validation & operational assessment of lab-free and high-throughput screening solutions (preprint)

BackgroundUncertainty in patients COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, typical turnaround times for batch-processed laboratory PCR tests remain 12-24h. Although rapid antigen lateral flow testing (LFD) has been widely adopted in UK emergency care settings, sensitivity is limited. We recently demonstrated that AI-driven triage (CURIAL-1.0) allows high-throughput COVID-19 screening using clinical data routinely available within 1h of arrival to hospital. Here we aimed to determine operational and safety improvements over standard-care, performing external/prospective evaluation across four NHS trusts with updated algorithms optimised for generalisability and speed, and deploying a novel lab-free screening pathway in a UK emergency department. MethodsWe rationalised predictors in CURIAL-1.0 to optimise separately for generalisability and speed, developing CURIAL-Lab with vital signs and routine laboratory blood predictors (FBC, U&E, LFT, CRP) and CURIAL-Rapide with vital signs and FBC alone. Models were calibrated during training to 90% sensitivity and validated externally for unscheduled admissions to Portsmouth University Hospitals, University Hospitals Birmingham and Bedfordshire Hospitals NHS trusts, and prospectively during the second-wave of the UK COVID-19 epidemic at Oxford University Hospitals (OUH). Predictions were generated using first-performed blood tests and vital signs and compared against confirmatory viral nucleic acid testing. Next, we retrospectively evaluated a novel clinical pathway triaging patients to COVID-19-suspected clinical areas where either model prediction or LFD results were positive, comparing sensitivity and NPV with LFD results alone. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser (OLO; SightDiagnostics, Israel) to provide lab-free COVID-19 screening in the John Radcliffe Hospitals Emergency Department (Oxford, UK), as trust-approved service improvement. Our primary improvement outcome was time-to-result availability; secondary outcomes were sensitivity, specificity, PPV, and NPV assessed against a PCR reference standard. We compared CURIAL-Rapides performance with clinician triage and LFD results within standard-care. Results72,223 patients met eligibility criteria across external and prospective validation sites. Model performance was consistent across trusts (CURIAL-Lab: AUROCs range 0.858-0.881; CURIAL-Rapide 0.836-0.854), with highest sensitivity achieved at Portsmouth University Hospitals (CURIAL-Lab:84.1% [95% Wilsons score CIs 82.5-85.7]; CURIAL-Rapide:83.5% [81.8 - 85.1]) at specificities of 71.3% (95% Wilsons score CIs: 70.9 - 71.8) and 63.6% (63.1 - 64.1). For 3,207 patients receiving LFD-triage within routine care for OUH admissions between December 23, 2021 and March 6, 2021, a combined clinical pathway increased sensitivity from 56.9% for LFDs alone (95% CI 51.7-62.0) to 88.2% with CURIAL-Rapide (84.4-91.1; AUROC 0.919) and 85.6% with CURIAL-Lab (81.6-88.9; AUROC 0.925). 520 patients were prospectively enrolled for point-of-care FBC analysis between February 18, 2021 and May 10, 2021, of whom 436 received confirmatory PCR testing within routine care and 10 (2.3%) tested positive. Median time from patient arrival to availability of CURIAL-Rapide result was 45:00 min (32-64), 16 minutes (26.3%) sooner than LFD results (61:00 min, 37-99; log-rank p<0.0001), and 6:52 h (90.2%) sooner than PCR results (7:37 h, 6:05-15:39; p<0.0001). Sensitivity and specificity of CURIAL-Rapide were 87.5% (52.9-97.8) and 85.4% (81.3-88.7), therefore achieving high NPV (99.7%, 98.2-99.9). CURIAL-Rapide correctly excluded COVID-19 for 58.5% of negative patients who were triaged by a clinician to COVID-19-suspected (amber) areas. ImpactCURIAL-Lab & CURIAL-Rapide are generalisable, high-throughput screening tests for COVID-19, rapidly excluding the illness with higher NPV than LFDs. CURIAL-Rapide can be used in combination with near-patient FBC analysis for rapid, lab-free screening, and may reduce the number of COVID-19-negative patients triaged to enhanced precautions ( amber) clinical areas.

Development and Validation of Early Warning Score Systems for COVID-19 Patients (preprint)

COVID-19 is a major, urgent, and ongoing threat to global health. Globally more than 24 million have been infected and the disease has claimed more than a million lives as of October 2020. Predicting which patients will need respiratory support is important to guiding individual patient treatment and also to ensuring sufficient resources are available. We evaluated the ability of six common Early Warning Scores (EWS) to identify respiratory deterioration defined as the need for advanced respiratory support (high-flow nasal oxygen, continuous positive airways pressure, non-invasive ventilation, intubation) within a prediction window of 24 hours. We show these scores perform sub-optimally at this specific task. Therefore, we develop an alternative Early Warning Score based on a Gradient Boosting Trees (GBT) algorithm that is able to predict deterioration within the next 24 hours with high AUROC 94% and an accuracy, sensitivity and specificity of 70%, 96%, 70%, respectively. Our GBT model outperformed the best EWS (LDTEWS:NEWS), increasing the AUROC by 14%. Our GBT model makes the prediction based on the current and baseline measures of routinely available vital signs and blood tests.